Sentiment Analysis in the Context of Disaster Responses
Contents
Sentiment Analysis in the Context of Disaster Responses#
“Requirements” for a Sentiment Analysis Methodology for Disaster Response#
To create a reliable “pipeline” that can be meaningful in the applied context of a disaster response:
An appropriate methodology
Which is the best methodology for our use-cases? What characteristics are to be prioritised to fit the needs?
An experimentation phase will likely be required. That phase will include both method and dataset testing
Open quesion: Use what is “out there” and construct a pipeline approach for disaster datasets (from input CSV to output spaital analyis) or research model optimisation? Probably a question of resources.
Training datasets
Scarce availability of disaster-related datasets
Option of “rehydrating” tweet_ids, but less than 50% are actually retrievable
Depending on aims: multi-lingual? English? Multimodal?
Own dataset annotation?
A “standardised” spatial and temporal analysis method should be assembled
Hotspot analysis
Trends over time
Fast output generation (template-based?)
(A Few) Methods#
Model comparisons should be expanded, but also purposefully chosen.
| Models | Precision | Recall | Accuracy | F1 | |
|---|---|---|---|---|---|
| 0 | SentiWordNet | 0.50 | 0.50 | 0.48 | 0.46 |
| 1 | VADER | 0.67 | 0.64 | 0.64 | 0.64 |
| 2 | Naive Bayes | 0.80 | 0.79 | 0.79 | 0.79 |
| 3 | SVM | 0.88 | 0.88 | 0.88 | 0.88 |
| 4 | RoBERTa | 0.71 | 0.73 | 0.71 | 0.71 |
| 5 | BERTweet | 0.71 | 0.72 | 0.70 | 0.71 |
| 6 | Fine Tuned BERT | 0.78 | 0.78 | 0.78 | 0.78 |
Datasets#
A start…
# |
Name |
Size |
Polarity |
Twitter Data |
Link |
|---|---|---|---|---|---|
1 |
Sentiment140 |
1.6m |
neg, neu, pos |
yes |
|
2 |
Twitter US Airline Sentiment |
55K |
neg, neu, pos |
yes |
|
3 |
SemEval-2015 Task 10: Sentiment Analysis in Twitter |
3097 |
neg, neu, pos |
yes |
|
4 |
SemEval-2016 Task 4: Sentiment Analysis in Twitter |
10K |
neg, neu, pos |
yes |
|
4 |
SemEval-2016 Task 5: Aspect-Based Sentiment Analysis |
10K |
entity, attribute, pos, neu, neg, target |
yes |
|
5 |
SemEval-2017 Task 4: Sentiment Analysis in Twitter |
neg, neu, pos |
yes |
||
6 |
SemEval-2018 Task 1: Affect in Tweets |
7K |
joy, fear, anger, sadness |
yes |
|
7 |
The Sanders Twitter Corpus |
5K |
neg, neu, pos, irrelevant |
yes |
|
8 |
The CrisisNLP dataset |
x million (various use cases) |
diverse |
yes |
|
9 |
TBCOV: Two Billion Multilingual COVID-19 Tweets with Sentiment, Entity, Geo, and Gender Labels |
2 b |
neg, neu, pos |
yes |
|
10 |
Natural Hazards Twitter Dataset |
40K |
pos, neg |
yes |
|
11 |
The TweetEval dataset |
||||
12 |
IMDb movie reviews dataset |
||||
13 |
Stanford Sentiment Treebank |
In everyday life…#
"Been trying to clear out my left ear with peroxide for the last 3 hours. This shit is bananas. I hate not being able to hear"
"Manchester United confirms that Carlos Tevez is leaving the club... so sad"
I wanna do something fun but have no clue what to do. Tre sleep & @krob5858 is taking a nap. So its just me
[Twitter Sentiment dataset with 30K tweets: see dataest]
| text | label | sentiment | |
|---|---|---|---|
| 2130 | just woke up, I`m starving | 0 | neutral |
| 11639 | four shots of novacaine in my mouth my right cheek is totally numb, booo. | -1 | negative |
| 15569 | Who`d have thought Wallace & Gromit and the team behind Monkey Island could have combined so disastrously? | -1 | negative |
| 8148 | Don`t know yet Lemme know if you come up with something though. | 0 | neutral |
| 18341 | flu or allergy??? ... Doesn`t matter, just try to squeeze my Sundayyyy | -1 | negative |
In a disaster…#
"Hope the people there are in a safe location and help reaches those in need during Hurricane Harvey. #prayfortexas"
"And the president pardons him on Friday during the prime of tropical storm Harvey."
"Our View: Hurricane Harvey proves we can still save each other"
"Still looking for an answer for power outages due to hurricane Harvey. I can have the best internet connection but no power = no sling"
There’s a scarcity of sentiment labelled twitter datasets. This one is derived from The Climate Change Twitter Dataset:
| date | text | |
|---|---|---|
| 47598 | Mon Aug 28 23:29:00 +0000 2017 | Hurricane Harvey leaves Texans without internet, phone service http://ift.tt/2xHXy6A� |
| 32134 | Sat Aug 26 00:51:00 +0000 2017 | On a day when the nation should be focusing on Hurricane Harvey, Trump decides to pardon a racist, convicted felon.Actions-Trump is racist! |
| 3945 | Fri Aug 25 02:44:00 +0000 2017 | Check out the latest on Hurricane Harvey and our Friday Football Forecast. https://www.41nbc.com/2017/08/24/harvey-threatens-texas-nice-weekend-store-middle-ga/�� |
| 26719 | Sat Aug 26 00:25:00 +0000 2017 | I added a video to a @YouTube playlist http://youtu.be/Y8as4cuFZ0A?a� LIVE COVERAGE: EXTREMELY DANGEROUS Category 4 Hurricane Harvey BATTERS |
| 13588 | Fri Aug 25 12:47:00 +0000 2017 | Food safety tips for Hurricane Harvey http://www.chron.com/news/houston-weather/hurricanes/article/Food-safety-tips-for-Hurricane-Harvey-Houston-11956465.php?cmpid=twitter-mobile�� via @houstonchron |
Some pos, neg datasets however are readily available. These are a few examples from the [Natural Hazards Twitter Dataset]('https://github.com/Dong-UTIL/Natural-Hazards-Twitter-Dataset'):
2017 Hurricane Harvey unlabelled dataset: 30000 tweets
2017 Hurricane Harvey labelled dataset (pos, neg): 7823 tweets
2018 Wildfires in the US labelled dataset (pos, neg): 4596 tweets
2013 floods in the US labelled dataset (pos, neg): 3597 tweets
Lexicon-Based Methods#
Lexicon-based approaches are still used in certain cases, especially when the goal is to perform sentiment analysis quickly and with a limited amount of computational resources. In lexicon-based sentiment analysis, words are assigned a sentiment score based on a pre-existing lexicon, such as SentiWordNet or the AFINN lexicon, and the overall sentiment of a text is calculated by summing the sentiment scores of the individual words.
Lexicon-based approaches have several advantages:
Ease of use
Speed
Interpretability
But there are downsides:
Difficulties with sarcasm, negation, sentiment of words in context
| Sentiment Score | |
|---|---|
| beautiful | 2.9 |
| beautifuler | 2.1 |
| beautifulest | 2.6 |
| beautifully | 2.7 |
| beautifulness | 2.6 |
| beautify | 2.3 |
| beautifying | 2.3 |
| beauts | 1.7 |
| beauty | 2.8 |
| belittle | -1.9 |
Lexicon: SentiWordNet#
SentiWordNet is built via a semi supervised method and could be a valuable resource for performing opinion mining tasks: it provides a readily available database of term sentiment information for the English language, and could be used as a replacement to the process of manually deriving ad-hoc opinion lexicons.
by nltk (see model info)
import nltk
nltk.download('sentiwordnet')
from nltk.corpus import sentiwordnet as swn
from nltk.tokenize import word_tokenize
def get_sentiment(tweet):
words = word_tokenize(tweet)
sentiment = 0.0
for word in words:
synset = list(swn.senti_synsets(word))
if len(synset) > 0:
sentiment += synset[0].pos_score() - synset[0].neg_score()
return sentiment
def classify_sentiment(tweet):
sentiment = get_sentiment(tweet)
if sentiment > 0:
return 1
elif sentiment < 0:
return -1
else:
return 0
# normal tweets for evaluation and comparison
sentiments1 = [classify_sentiment(tweet) for tweet in normal_tweets['text']]
normal_tweets['nltk_sentiwordnet'] = sentiments1
# disaster tweets for comparison
sentiments2 = [classify_sentiment(tweet) for tweet in harvey_data['text']]
harvey_data['nltk_sentiwordnet'] = sentiments2
| Precision | Recall | Accuracy | F1 | |
|---|---|---|---|---|
| SentiWordNet | 0.5 | 0.5 | 0.48 | 0.46 |
Lexicon: VADER (Valence Aware Dictionary and sEntiment Reasoner)#
VADER is specifically attuned to sentiments expressed in social media. It uses a combination of sentiment-related words and emojis, along with some simple heuristics (punctuation, capitalisation, degree modifiers, conjuctions), to assign a sentiment score (positive, negative, or neutral) to a given piece of text. It’s output sentiment score is a numeric score between -1 and +1. The word sentiment scores range from -4 to 4 (neg to pos).
by Hutto & Gilbert see model info
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer # VADER
sa = SentimentIntensityAnalyzer()
scores = {}
listscores = []
for tweet in normal_tweets['text']:
scores[sa.polarity_scores(tweet)['compound'] ] = str(tweet)
listscores.append(sa.polarity_scores(tweet)['compound'])
normal_tweets['vader'] = listscores
scores1 = {}
listscores1 = []
for tweet in harvey_data['text']:
scores1[sa.polarity_scores(tweet)['compound'] ] = str(tweet)
listscores1.append(sa.polarity_scores(tweet)['compound'])
harvey_data['vader'] = listscores1
normal_tweets['vader'] = [1 if value >= 0.3 else -1 if value <= -0.3 else 0 for value in normal_tweets['vader']]
harvey_data['vader'] = [1 if value >= 0.3 else -1 if value <= -0.3 else 0 for value in harvey_data['vader']]
performance = calc_error_matrices(normal_tweets, 'VADER', 'label', 'vader')
performance
| Precision | Recall | Accuracy | F1 | |
|---|---|---|---|---|
| SentiWordNet | 0.50 | 0.50 | 0.48 | 0.46 |
| VADER | 0.67 | 0.64 | 0.64 | 0.64 |
Machine Learning Method: Naive Bayes#
Naive Bayes is a probabilistic algorithm (based on Bayes’ theorem). It uses the probability of words or terms appearing in documents of different categories to determine the likelihood that a new document belongs to each category. It involves two basic steps:
Training: The algorithm learns the probability of words or terms appearing in each category. This is simply done by counting the number of occurrences of each word or term in the training corpus for each category and then computing the probability of each word or term given the category. This results in a set of word probabilities for each category.
Classification: The algorithm uses the probabilities learned in the training step to classify new documents. For a new document, the algorithm calculates the likelihood of the document being in each category based on the probabilities of its words or terms. The category with the highest likelihood is chosen as the classification for the document.
The “naive” aspect of Naive Bayes comes from the assumption that the occurrences of words or terms in a document are independent of one another, which is not always true(!).
by nltk see model info
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import accuracy_score
# split the data into training and testing sets
train_text, test_text, train_labels, test_labels = train_test_split(normal_tweets['text'], normal_tweets['label'], random_state=42)
# create a bag of words representation of the text data
vectorizer = CountVectorizer()
train_text_vectors = vectorizer.fit_transform(train_text)
test_text_vectors = vectorizer.transform(test_text)
# fit a Naive Bayes classifier on the training data
clf = MultinomialNB()
clf.fit(train_text_vectors, train_labels)
text_vectors = vectorizer.transform(normal_tweets['text'])
# make predictions on the text data using the trained classifier
predictions = clf.predict(text_vectors)
# add the predictions as a new column in the dataframe
normal_tweets['naive_bayes'] = predictions
# same for disaster data
text_vectors1 = vectorizer.transform(harvey_data['text'])
predictions1 = clf.predict(text_vectors1)
# add the predictions as a new column in the dataframe
harvey_data['naive_bayes'] = predictions1
plotting('naive_bayes', 'Naive Bayes')
| Precision | Recall | Accuracy | F1 | |
|---|---|---|---|---|
| SentiWordNet | 0.50 | 0.50 | 0.48 | 0.46 |
| VADER | 0.67 | 0.64 | 0.64 | 0.64 |
| Naive Bayes | 0.80 | 0.79 | 0.79 | 0.79 |
Machine Learning Method: Support Vector Machine#
Using readily available libraries like sklearn, an SVM classifier (in this case SVC (Support Vector Classification)) can be trained for binary classification problems like pos-neg sentiment classification. The SVC classifier predicts the class label of a given sample based on the feature set. It solves the optimization problem to find a hyperplane that maximally separates the positive and negative class samples in the feature space. The optimization problem is solved using the maximum margin principle, where the margin is the distance between the hyperplane and the closest samples from each class, called support vectors. The SVC classifier is a useful tool for performing binary classification problems, particularly when the number of features is high, and the data is not linearly separable.
by sklearn see model info
# Import libraries
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
from sklearn import svm
# split the data into training and testing sets
train_text, test_text, train_labels, test_labels = train_test_split(normal_tweets['text'], normal_tweets['label'], random_state=42)
# create a bag of words representation of the text data
vectorizer = CountVectorizer()
train_text_vectors = vectorizer.fit_transform(train_text)
test_text_vectors = vectorizer.transform(test_text)
# fit an SVM classifier on the training data
clf = SVC(kernel='linear')
clf.fit(train_text_vectors, train_labels)
# Test the classifier
X_new = vectorizer.transform(normal_tweets["text"])
y_pred = clf.predict(X_new)
normal_tweets['svm'] = y_pred
X_new1 = vectorizer.transform(harvey_data["text"])
y_pred1 = clf.predict(X_new1)
harvey_data['svm'] = y_pred1
performance = calc_error_matrices(normal_tweets, 'SVM', 'label', 'svm')
performance
| Precision | Recall | Accuracy | F1 | |
|---|---|---|---|---|
| SentiWordNet | 0.50 | 0.50 | 0.48 | 0.46 |
| VADER | 0.67 | 0.64 | 0.64 | 0.64 |
| Naive Bayes | 0.80 | 0.79 | 0.79 | 0.79 |
| SVM | 0.88 | 0.88 | 0.88 | 0.88 |
Deep Learning Models#
In recent years, the use of lexicon-based methods and traditional machine learning models for sentiment analysis has decreased in popularity. This shift has been largely driven by the emergence of deep learning models, which have shown impressive results on NLP tasks such as sentiment classification. With the availability of vast amounts of text data, deep learning models such as transformers have been able to capture more complex patterns and relationships within the data, leading to improved performance compared to lexicon-based and machine learning models. As a result, deep learning methods are now seen as the state-of-the-art approach for sentiment analysis and other NLP tasks.
Recurrent Neural Network#
A Recurrent Neural Network is a generalization of feedforward neural network that has an “internal memory”. An RNN is recurrent in nature as it performs the same function for every input of data while the output of the current input depends on the past one computation. After producing the output, it is copied and sent back into the recurrent network. For making a decision, it considers the current input and the output that it has learned from the previous input.
model = ltsm_model((maxLen,), word_to_vec_map, word_to_index)
model.summary()
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 29)] 0
embedding_1 (Embedding) (None, 29, 50) 20000050
lstm (LSTM) (None, 29, 128) 91648
dropout (Dropout) (None, 29, 128) 0
lstm_1 (LSTM) (None, 128) 131584
dropout_1 (Dropout) (None, 128) 0
dense_1 (Dense) (None, 3) 387
activation (Activation) (None, 3) 0
=================================================================
Total params: 20,223,669
Trainable params: 223,619
Non-trainable params: 20,000,050
_________________________________________________________________
Gated Recurrent Units#
A Gated Recurrent Unit (GRU) is a type of Recurrent Neural Network (RNN) that is used to process sequential data. It is designed to handle the vanishing gradient problem that can occur when training traditional RNNs by incorporating gating mechanisms.
# Defining GRU model
gru_model = Sequential(name="GRU_Model")
gru_model.add(Embedding(vocab_size,
embd_len,
input_length=max_words))
gru_model.add(GRU(128,
activation='tanh',
return_sequences=False))
gru_model.add(Dense(1, activation='sigmoid'))
# Printing the Summary
print(gru_model.summary())
# Compiling the model
gru_model.compile(
loss="binary_crossentropy",
optimizer='adam',
metrics=['accuracy']
)
# Training the GRU model
history2 = gru_model.fit(x_train_, y_train_,
batch_size=64,
epochs=5,
verbose=1,
validation_data=(x_valid, y_valid))
# Printing model score on test data
print()
print("GRU model Score---> ", gru_model.evaluate(x_test, y_test, verbose=0))
Long Short Term Memory Network#
LSTM (Long Short-Term Memory) is a type of Recurrent Neural Network (RNN) that is designed to overcome the problem of vanishing gradients in traditional RNNs by introducing a gating mechanism that controls the flow of information within the network. An LSTM network contains a series of memory cells that are responsible for remembering the information from earlier time steps. There are three gates in an LSTM cell that control the flow of information: the input gate, the forget gate, and the output gate. These gates allow the LSTM to selectively decide which information to forget and which information to pass to the output.
# Defining LSTM model
lstm_model = Sequential(name="LSTM_Model")
lstm_model.add(Embedding(vocab_size,
embd_len,
input_length=max_words))
lstm_model.add(LSTM(128,
activation='relu',
return_sequences=False))
lstm_model.add(Dense(1, activation='sigmoid'))
# Printing Model Summary
print(lstm_model.summary())
# Compiling the model
lstm_model.compile(
loss="binary_crossentropy",
optimizer='adam',
metrics=['accuracy']
)
# Training the model
history3 = lstm_model.fit(x_train_, y_train_,
batch_size=64,
epochs=5,
verbose=2,
validation_data=(x_valid, y_valid))
# Displaying the model accuracy on test data
print()
print("LSTM model Score---> ", lstm_model.evaluate(x_test, y_test, verbose=0))
Transformer-Based Models#
In recent years, the transformer model has revolutionized the field of NLP. This ‘new’ deep learning approach has been highly successful in a variety NLP tasks, including sentiment analysis. The transformer model offers several advantages over traditional machine learning and even other deep learning approaches and have been shown to outperform traditional machine learning and other deep learning methods on NLP tasks, particularly sentiment analysis. Some of the key advantages it has are:
The encoder-decoder framework: Encoder generates a representation of the input (semantic, context, positional) and the decoder generates output. Common use case: sequence to sequence translation tasks.
Attention mechanisms: Deals with the information bottleneck of the traditional encoder-decoder architecture (where one final encoder hidden state is passed to decoder) by allowing the decoder to access the hidden states at each step and being able to prioritise which state is most relevant.
Transfer learning (i.e. fine-tuning a pre-trained language model)
Transformer-Based Model: RoBERTa#
RoBERTA (Robustly Optimized BERT Pretraining Approach) has the same architecture as BERT but marks an improved version of BERT for several reasons:
RoBERTa was trained on 10x as much data as was used for BERT training (160GB, compared to 16GB for BERT)
Dynamic masking was used during training, rather than fixed masking in BERT
the next sentence prediction was left out during training, which is arguably not essential especially when considering tweets.
by cardiffnlp see model info
# apply in the form of a function so it can be called for usecase later on
def robertabase_apply(dataset):
# create variable for labels (good to bad)
labels= ['positive', 'neutral', 'negative']
# lists to be filled
cardiffroberta_sentiment_prediction = []
cardiffroberta_sentiment_prediction_softmax = []
cardiffroberta_sentiment_prediction_num = []
# iterate over dataset
for index, row in dataset.iterrows():
text = row['text']
text = preprocess(text)
encoded_input = tokenizer_tw_rob_base_sent_lat(text, truncation=True, max_length=500, return_tensors='pt')
output = model_tw_rob_base_sent_lat(**encoded_input)
score = np.round(softmax(output[0][0].detach().numpy()), 4)
label = config_tw_rob_base_sent_lat.id2label[np.argsort(score)[::-1][0]]
cardiffroberta_sentiment_prediction.append(label)
cardiffroberta_sentiment_prediction_softmax.append(max(score))
# positive label
if label == labels[0]:
cardiffroberta_sentiment_prediction_num.append(1)
# negative label
elif label == labels[2]:
cardiffroberta_sentiment_prediction_num.append(-1)
# neutral label
else:
cardiffroberta_sentiment_prediction_num.append(0)
#dataset['cardiffroberta_sentiment_prediction'] = cardiffroberta_sentiment_prediction
#dataset['cardiffroberta_sentiment_prediction_softmax'] = cardiffroberta_sentiment_prediction_softmax
dataset['cardiffroberta'] = cardiffroberta_sentiment_prediction_num
model_name = "cardiffroberta"
# model name and labels will be needed later on as input variables for plotting and mapping
print("Variables that will later be required for plotting and mapping:")
return model_name, labels
| Precision | Recall | Accuracy | F1 | |
|---|---|---|---|---|
| SentiWordNet | 0.50 | 0.50 | 0.48 | 0.46 |
| VADER | 0.67 | 0.64 | 0.64 | 0.64 |
| Naive Bayes | 0.80 | 0.79 | 0.79 | 0.79 |
| SVM | 0.88 | 0.88 | 0.88 | 0.88 |
| RoBERTa | 0.71 | 0.73 | 0.71 | 0.71 |
Transformer-based Model: BERTweet#
This is a BERTweet-base RoBERTa model trained on SemEval 2017 (~40k Tweets). It uses POS, NEG, NEU labels and is suitable for English and Spanish languages. pysentimiento is an open-source library for non-commercial use and scientific research purposes only. Please be aware that models are trained with third-party datasets and are subject to their respective licenses.
by finiteautomata see model info
# apply in the form of a function so it can be called for usecase later on
def bertweet_apply(dataset):
# create labels variable
labels = ['POS', 'NEU', 'NEG']
# lists to be filled
bertweet_sentiment_prediction = []
bertweet_sentiment_prediction_softmax = []
bertweet_sentiment_prediction_num = []
# iterate over dataframe
for index, row in dataset.iterrows():
text = row['text']
text = bertweetpreprocess(text)
result = bertweetanalyzer.predict(text)
label = result.output
bertweet_sentiment_prediction.append(label)
bertweet_sentiment_prediction_softmax.append(np.round(result.probas[label], 4))
if label == labels[0]:
bertweet_sentiment_prediction_num.append(1)
elif label == labels[2]:
bertweet_sentiment_prediction_num.append(-1)
else:
bertweet_sentiment_prediction_num.append(0)
#dataset['bertweet_sentiment_prediction'] = bertweet_sentiment_prediction
#dataset['bertweet_sentiment_prediction_softmax'] = bertweet_sentiment_prediction_softmax
dataset['bertweet'] = bertweet_sentiment_prediction_num
model_name = "bertweet"
# model name and labels will be needed later on as input variables for plotting and mapping
print("Variables that will later be required for plotting and mapping:")
return model_name, labels
| Precision | Recall | Accuracy | F1 | |
|---|---|---|---|---|
| SentiWordNet | 0.50 | 0.50 | 0.48 | 0.46 |
| VADER | 0.67 | 0.64 | 0.64 | 0.64 |
| Naive Bayes | 0.80 | 0.79 | 0.79 | 0.79 |
| SVM | 0.88 | 0.88 | 0.88 | 0.88 |
| RoBERTa | 0.71 | 0.73 | 0.71 | 0.71 |
| BERTweet | 0.71 | 0.72 | 0.70 | 0.71 |
Fine-Tuned Downstream Sentiment Analysis#
This is a BERT base model (uncased), pretrained on English language using a masked language modeling (MLM) objective. This model is uncased: it does not make a difference between english and English.
This is a fine-tuned downstream version of the bert-base-uncased model for sentiment analysis, this model is not intended for further downstream fine-tuning for any other tasks. This model is trained on a classified dataset for text classification.
by seethal see model info
# apply in the form of a function so it can be called for usecase later on
def seethal_gen_data_apply(dataset):
# create variable for labels (good to bad)
labels= ['positive', 'neutral', 'negative']
# lists to be filled
seethal_gen_data_sentiment_prediction = []
seethal_gen_data_sentiment_prediction_softmax = []
seethal_gen_data_sentiment_prediction_num = []
# iterate over dataset
for index, row in dataset.iterrows():
text = row['text']
text = preprocess(text)
encoded_input = tokenizer_seethal_gen_data(text, truncation=True, max_length=500, return_tensors='pt')
output = model_seethal_gen_data(**encoded_input)
score = np.round(softmax(output[0][0].detach().numpy()), 4)
label = config_tw_rob_base_sent_lat.id2label[np.argsort(score)[::-1][0]]
seethal_gen_data_sentiment_prediction.append(label)
seethal_gen_data_sentiment_prediction_softmax.append(max(score))
# positive label
if label == labels[0]:
seethal_gen_data_sentiment_prediction_num.append(1)
# negative label
elif label == labels[2]:
seethal_gen_data_sentiment_prediction_num.append(-1)
# neutral label
else:
seethal_gen_data_sentiment_prediction_num.append(0)
#dataset['seethal_gen_data_sentiment_prediction'] = seethal_gen_data_sentiment_prediction
#dataset['seethal_gen_data_sentiment_prediction_softmax'] = seethal_gen_data_sentiment_prediction_softmax
dataset['finetunedBERT'] = seethal_gen_data_sentiment_prediction_num
model_name = "seethal_gen_data"
# model name and labels will be needed later on as input variables for plotting and mapping
print("Variables that will later be required for plotting and mapping:")
return model_name, labels
| Precision | Recall | Accuracy | F1 | |
|---|---|---|---|---|
| SentiWordNet | 0.50 | 0.50 | 0.48 | 0.46 |
| VADER | 0.67 | 0.64 | 0.64 | 0.64 |
| Naive Bayes | 0.80 | 0.79 | 0.79 | 0.79 |
| SVM | 0.88 | 0.88 | 0.88 | 0.88 |
| RoBERTa | 0.71 | 0.73 | 0.71 | 0.71 |
| BERTweet | 0.71 | 0.72 | 0.70 | 0.71 |
| Fine Tuned BERT | 0.78 | 0.78 | 0.78 | 0.78 |
Data Visualisations#
gdf_harvey.head()
| TWEET_ID | DATE | text | LAT | LONG | MATCHING_I | TOPIC_NUMB | PROBABILIT | WEIGHT | geometry | bertweet | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 9.058685e+17 | 2017-09-07 19:00:49 | Two more days!!! https://t.co/vydsOgG5RY | 30.223722 | -97.744334 | 29 | 0 | 0.066667 | 1 | POINT (620838.338 3344242.541) | 1 |
| 1 | 9.041387e+17 | 2017-09-03 00:27:26 | Baton Rouge: 7:24pm: sunset | 30.440000 | -91.190000 | 212 | 0 | 0.666367 | 1 | POINT (1251048.695 3393577.497) | 0 |
| 2 | 9.057606e+17 | 2017-09-07 11:52:06 | Be Prepared! Stay Prepared! @ Fire Station 93 ... | 29.590136 | -95.180497 | 17 | 8 | 0.666367 | 1 | POINT (870027.953 3279466.422) | 1 |
| 3 | 9.010665e+17 | 2017-08-25 12:59:26 | #hurricaneharvey @ Houston, Texas https://t.co... | 29.762900 | -95.383200 | 1972 | 1 | 0.999534 | 1 | POINT (849776.868 3297998.151) | 0 |
| 4 | 9.058019e+17 | 2017-09-07 14:36:20 | Just posted a photo @ Ruff Cuts & PurrFect... | 30.228525 | -92.658851 | 20 | 1 | 0.999650 | 1 | POINT (1110801.489 3361168.665) | 1 |
Ratios between pos, neu, and neg sentiments: [4891, 21570, 2472]
Country USA was either not found or too many matches found in centroids dataframe. Defaulting to USA.